Jul 19
3
With Google aiming to make the robots.txt exclusion protocol a standard, they proposed some changes and submitted them the other day. Now, Google updated their own developer docs around the robots.txt specification to match
Continued here:
List Of All The GoogleBot Robots.txt Specifications Changes